Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex-ante and ex-post peer evaluation data from the Austrian science fund

It is essential for research funding organizations to ensure both the validity and fairness of the grant approval procedure. The ex-ante peer evaluation (EXANTE) of N = 8,496 grant applications submitted to the Austrian Science Fund from 1999 to 2009 was statistically analyzed. For 1,689 funded research projects an ex-post peer evaluation (EXPOST) was also available; for the rest of the grant applications a multilevel missing data imputation approach was used to consider verification bias for the first time in peer-review research. Without imputation, the predictive validity of EXANTE was low (r = .26) but underestimated due to verification bias, and with imputation it was r = .49. That is, the decision-making procedure is capable of selecting the best research proposals for funding. In the EXANTE there were several potential biases (e.g., gender). With respect to the EXPOST there was only one real bias (discipline-specific and year-specific differential prediction). The novelty of this contribution is, first, the combining of theoretical concepts of validity and fairness with a missing data imputation approach to correct for verification bias and, second, multilevel modeling to test peer review-based funding decisions for both validity and fairness in terms of potential and real biases. It is important for a research funding organization to ensure the quality of its peer review-based decision-making procedure. On behalf of the Austrian Science Fund (FWF), Austria’s leading funding organization for basic research, external meta-evaluations of its grant approval procedures were recurrently conducted (Dinges, 2005). For a metaevaluation of a research funding organization’s decisionmaking procedure, the following three questions are essential (Bornmann & Daniel, 2005; Daniel, 2004): (a) Is the procedure reliable—that is, what is the extent of agreement among reviewers on the quality of a grant application and on whether the proposed research project should be funded or not (ex-ante peer evaluation, EXANTE)?; (b) Is the procedure valid—that is, do peer reviewers’ overall ratings (EXANTE) or the funding decisions of the Board of Trustees correlate with scientific performance measures subsequent to the funding decisions (ex-post peer evaluation, EXPOST)? Is the research funding organization capable of selecting the best grant proposals (EXANTE) with the best performance for funding (EXPOST)? (Irvine & Martin, 1984; Van den Besselaar & Leydesdorff, 2009); and (c) Is the procedure fair—that is, are there external factors that have an effect on the decision-making procedure but that have nothing to do with the quality of a grant proposal or the performance of a research project? The reliability of reviewers’ evaluations (EXANTE) at the FWF has been examined previously (Mutz, Bornmann, & Daniel, 2012b). In this study we focus on predictive Received November 27, 2013; Revised February 18, 2014; Accepted April